Online learning on a continuum

We study a sequential decision problem on a subset S ⊂ R. A decision maker chooses, on iteration t, a probability distribution π over S, then discovers a bounded loss function ` : S → [0,M ], and incurs the expectation Es∼π(t) `(s). The cumulative regret of the decision maker is then ∑T t=1 Es∼π(t) [`(x)] − infs∈S ∑t τ=1 ` (s). We investigate conditions under which one can guarantee a sublinear regret. Previous studies consider the case where S is convex: if the losses are convex, then a simple gradient descent algorithm guarantees a O( √ t) regret, and if the losses are exp-concave, a generalized Hedge algorithm guarantees a O(log t) regret. Building on this previous work, we relax the convexity assumption on S, and propose a generalized Hedge algorithm with a O( √ t log t) bound on the regret when the losses are Lipschitz (uniformly in time) and S is uniformly fat (a weaker condition than convexity). We compare our method to working with a finite cover of the set. In particular, we show that both guarantee a O( √ t log t) bound on the regret, but our method does not need to explicitly compute a cover.